Papers with n-gram overlap
Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Question generation models are often evaluated with standardized NLG metrics that are based on n-gram overlap. |
| Approach: | They propose to use QGen to help teachers automate the generation of reading comprehension quizzes by comparing n-gram overlap with BLEU to compare system-generated questions with heldout human-written references. |
| Outcome: | The best model had only 68.4% of its questions accepted by the ten teachers who participated in the study. |
Improving Abstraction in Text Summarization (D18-1)
Copied to clipboard
| Challenge: | Abstractive text summarization models do not capture the abstractive nature of high quality summaries. |
| Approach: | They propose to decompose a decoder into a contextual network and a pretrained language model that incorporates prior knowledge about language generation. |
| Outcome: | The proposed model achieves comparable results to state-of-the-art models, based on ROUGE scores and human evaluations, while producing a significantly higher level of abstraction. |
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent generative language models have shown promise in abstractive summarization tasks. |
| Approach: | They propose to use Fr echet embedding distance and angular embeddable similarity to evaluate the performance of generative language models in abstractive summarization tasks. |
| Outcome: | The proposed metric shows close relation with human judgments and has overall better correlations with them. |
Automatic Readability Assessment for Closely Related Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | In recent years, the main focus of research on automatic readability assessment (ARA) has shifted towards using expensive deep learning-based methods with the primary goal of increasing models’ accuracy. |
| Approach: | They focus on how linguistic aspects such as mutual intelligibility or degree of language relatedness can improve ARA in a low-resource setting. |
| Outcome: | The inclusion of CrossNGO, a novel feature exploiting n-gram overlap, significantly improves the performance of ARA models compared to the use of off-the-shelf large multilingual language models alone. |
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are slow and abstractive, but we propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries. |
| Approach: | They propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries by exposing diverse pseudo summary with two supervision to the student model. |
| Outcome: | The proposed method outperforms previous methods in abstractive summarization distillation, producing highly abstractive and informative summaries. |
Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization (2020.coling-main)
Copied to clipboard
Dongyub Lee, Myeong Cheol Shin, Taesun Whang, Seungwoo Cho, Byeongil Ko, Daniel Lee, EungGyun Kim, Jaechoon Jo
| Challenge: | Existing methods for text summarization are based on recall-oriented understudy for gisting evaluation (ROUGE) scores do not reflect semantic meaning correspondences between generated and reference summaries. |
| Approach: | They propose to use Korean as a summarization language to generate a shorter form of text from the source document preserving salient information. |
| Outcome: | The proposed evaluation metrics improve the correlation between the metrics and human judgment. |
In-context Examples Selection for Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative models can perform a wide range of NLP tasks using in-context learning. |
| Approach: | They aim to understand the properties of good in-context examples for machine translation in both in-domain and out-of-domain settings. |
| Outcome: | The proposed model outperforms a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets. |